Method for assisting analysis of production process, program for making computer execute that method, program product, and storage medium

ABSTRACT

A method of assisting analysis of a production process assisting work for analyzing the relationship between quality factors and characteristic of a product and a program product and a recording medium for the same, comprising receiving a designation of a factor from a user at a computer, arranging images corresponding to said image data related to the received factor in a virtual space in a display device connected to the computer (S 13 ), displaying the virtual space in which the images are arranged in the display device (S 14 ), repeating the designation reception, arrangement, and display until the user judges that there is similarity between adjoining images in the displayed images, receiving designation of at least one image by the user from images having similarity between adjoining images (S 15 ), automatically extracting common factors among factors of the designated image (S 16 ), receiving a hypothesis of a relationship between the common factors designated by the user and quality of products, and verifying the relation hypothesis (S 17 ) so as to thereby assist work for analyzing the relationship between factors selected by the user and the product characteristic and adding to factors feature quantity from the image data of products and feature quantity from text data relating to the process prepared by a production worker converted to numerals/characters (S 11 , S 12 ).

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Patent ApplicationNo. PCT/JP03/05635, filed on May 2, 2003, the contents beingincorporated herein by reference.

TECHNICAL FIELD

The present invention relates to a method for assisting analysis of aproduction process able to be utilized for analysis of a productionprocess for various types of industrial products in various industries,a program for making a computer execute this method, and a programproduct and storage medium, more particularly relates to a method forassisting analysis of a production process which assists analysis workof the relationship between factors which may influence the quality ofproducts and the quality/characteristics when the characteristics of afinished product or the characteristics of an intermediate product(hereinafter referred to as a “product”) are expressed as image data inthe process of production of a semiconductor, the process of productionof a magnetic disk device, the process of production of a display, etc.and a program, program product, and storage medium for the same.

BACKGROUND ART

In the processes for production of various industrial products, thereare numerous factors which may influence the quality/characteristics ofthe products. The interaction of these factors is not well understood.As such factors, for example, there are the material characteristics,the apparatus, temperature, pressure, current, viscosity, speed,humidity, air pressure, etc. Among these factors, there are factorshaving parameters (control factors) able to be controlled from theoutside and parameters which cannot be controlled (error factors).“Analysis of a production process” means the work for clarifying therelationship between these factors and product characteristics for thepurpose of improving the quality of the products.

In the past, due to the difficulty in measuring the factors, productionprocesses were analyzed based on the experience and intuition of skilledworkers. In recent years, however, the improvement in performance ofmeasuring instruments and data processing devices has made it possibleto easily obtain data relating to these factors or productcharacteristics. Therefore, as an approach for analyzing a productionprocess systematically without relying on experience or intuition,multivariate analysis and other statistical techniques (see TakaoMaruyama and Masaya Miyakawa, SQC Theory and Practice, Series“Mathematical Principles for the Contemporary Man”, Asakura Shoten(1992)) and data mining techniques are being used for process analysisin an increasing number of cases (see Hidetaka Tsuda, Hideo Shirai, andRiichiro Take, “Application of Data Mining to Yield Analysis”, 3rd DataMining Workshop Papers, Japan Society of Software Science Data MiningResearch Group (2002) and Michael J. A. Berry and Gordon Linoff,Mastering Data Mining, John Wiley & Sons Inc. (2000)).

In the conventional methods of analysis of production processes usingfactor analysis or other statistical techniques or data miningtechnology, the extents of the influence of explanatory variables on acriterion variable are statistically analyzed using various factors asexplanatory variables and using an indicator of quality as a criterionvariable. However, this has the following problems:

(a) The data obtained in a production process often includes severalhundred to several thousand of factors, while the number of recordswhich correspond to the number of lots is generally small. Therefore,with just the conventional statistical techniques, it is difficult toobtain significant results if the number of records is small.

(b) Even if image data expressing characteristics of a finished productor characteristics of an intermediate product or text data relating tothe process such as comments of a production worker are obtained in aproduction process, these are only treated as reference data and cannotbe directly utilized for process analysis. For example, sometimes aproduction worker will view the appearance of an intermediate product orfinished product and intuitively judge what parts of the process are inwhat states based on his previous experience, but this type of analysisis not possible with conventional statistical techniques.

(c) An inspector can visually inspect a product or utilize imagerecognition technology to convert appearance to numerical features forstatistical process analysis. However, converting slight changes inappearance to numerical features is difficult and the conversion maycause a loss of the information for process analysis.

Nonpatent Documents

[1] Takao Maruyama and Masaya Miyakawa, SQC Theory and Practice, Series“Mathematical Principles for the Contemporary Man”, Asakura Shoten(1992)

[2] Hidetaka Tsuda, Hideo Shirai, and Riichiro Take, “Application ofData Mining to Yield Analysis”, 3rd Data Mining Workshop Papers, JapanSociety for Software Science and Technology Data Mining Research Group(2002)

[3] Michael J. A. Berry and Gordon Linoff, Mastering Data Mining, JohnWiley & Sons Inc. (2000)

[4] Kohei Murao, “Automatic Extraction of Image Feature Quantities andSimilar Image Search”, Humanities and Data Processing, vol. 28, pp. 54to 61, Bensei Shuppan, July (2000).

http://venus/netlaboratory/com/salon/chiteki/mur/img search.html

[5] Vittorio Castelli and Lawrence D. Bergman ed.; Image Databases:Search and Retrieval of Digital Imagery, pp. 285 to 372, John Wiley &Sons (2002).

[6] Nishio et al., Data Structuring and Searching, pp. 113 to 119,Iwanami Shoten (2000).

[7] Haruo Yanagii, Multivariate Data Analysis Methods—Theory andApplication, Asakura Shoten (1994).

[8] Kohonen (translated by Takaya Toku): Self Organization Map, SpringerVerlag Tokyo (1996).

DISCLOSURE OF INVENTION

An object of the present invention is to solve the above problems of theconventional methods of analysis of production processes by providing amethod of assisting analysis of a production process based on the ideaof arranging and displaying images based on selected factors, thenreselecting factors and rearranging images until adjoining images of aplurality of products become similar, after that automaticallyextracting factors common to designated adjoining images, which assiststhe streamlining of work for analysis of the relationship betweenfactors and quality/characteristics without relying only on statisticaltechniques even when the number of records of products is small and aprogram, program product, and storage medium for the same.

Another object is to provide a method of assisting analysis of aproduction process assisting the streamlining of the work for analysisof the relationship between factors and quality/characteristics even ifthe person analyzing the production process is not a skilled worker anda program and storage medium for the same.

Still another object is to provide a method of assisting analysis of aproduction process for assisting streamlining of work for analyzing therelationship between factors and quality/characteristics by making animage of a product or text relating to the process prepared by aproduction worker part of the factors converted to numerals/charactersand a program, program product, and storage medium for the same.

To achieve the above objects, according to a first aspect of the presentinvention, there are provided a method of assisting analysis of aproduction process assisting work for analyzing the relationship betweena factor which may influence the quality of a product in a productionprocess and which is comprised of numeral/character data and aquality/characteristic of a product and a program, program product, andstorage medium storing the program. This method is provided with anarrangement routine arranging images based on designated factorsreceived from a user, a display routine displaying images based on thearrangement in a virtual space in a display device, a routine repeatingreception of designation of a factor from a user until the user judgesthat there is similarity between adjoining images in the displayedimages, an image designation reception routine receiving designations ofa plurality of images by the user from images having similarity betweenadjoining images, a common factor extraction routine automaticallyextracting common factors among factors of designated images, a relationhypothesis reception routine receiving a hypothesis of a relationshipbetween a factor and a quality of products, and a routine verifying therelation hypothesis. Due to this, the work of a user analyzing therelationship between selected factors and quality/characteristic isassisted.

According to the first aspect, an assisting environment is providedenabling a person conducting an analysis to perform analysis with theinclusion of his knowhow and experience while viewing data, so the smallnumber of records is made up for.

Specifically, the image data is made to be able to be directly handledby process analysis, so a user (that is, a person analyzing theproduction process) can view images or rearrange the images based onvarious viewpoints and thereby the proposal of a hypothesis relating tothe relationship between factors which may influence the quality ofproducts and the quality/characteristic of products is assisted.Further, it becomes easy to perform the verification work for confirmingto what extent a proposed hypothesis is reliable.

Further, in the first aspect, preferably there is further provided aroutine extracting from image data and text data relating to the processprepared by a production worker and expressing a quality/characteristicsof products feature quantities of the same converted tonumerals/characters and adding them to the factors, and the commonfactor extraction routine includes a routine extracting at least one ofthe image features, words, and factors common to selected images.

Due to this, it is possible to simply extract features common to imagesor text, so analysis of the production process of products becomeseasier.

Further, in the first aspect, the designated image reception routine mayinclude a routine receiving specific images designated by a user in avirtual space, and the display routine may include a routine listing upand displaying on a display means images similar to the designated imagewhen a user designates at least one of feature quantity or numeral of adesignated image and text data.

Due to this, images similar to a designated image can be easilyextracted, so analysis of the production process of products becomeseasier.

Further, in the first aspect, the routine verifying a hypothesizedrelationship may include a routine verifying a relationship between afactor and a quality/characteristics which a user finds.

Due to this, the relationship between a factor and aquality/characteristics can be reliably grasped.

Further, in the first aspect, the arrangement routine may include aroutine utilizing a self-organizing map.

Due to this, it is possible to display an image corresponding to fourdimensions or more of factors in a three-dimensional space, wherebyanalysis of the production process of products becomes easier.

Further, in the first aspect, the display routine includes a routinechanging a viewing point in a virtual space so as to assist the work ofviewing an image in a virtual space.

Due to this, it is possible for a user to freely view the inside of thevirtual space, so he can easily obtain a grasp of the relationshipbetween a desired factor and the quality/characteristics.

Further, in the first aspect, the display routine may include a routinedisplaying a virtual space at a display device connected to anothercomputer through a network.

Due to this, it is possible to display the virtual space at any locationutilizing a network in a client-server relationship.

BRIEF DESCRIPTION OF DRAWINGS

These objects and features of the present invention will become clearerfrom the best modes for working the invention described below withreference to the drawings.

FIG. 1 is a flow chart explaining in brief the method of assistinganalysis of a production process according to an embodiment of thepresent invention.

FIG. 2 is a flow chart explaining a method of extracting featurequantities converted to numerals/characters from images and text inprocess data shown in FIG. 1.

FIG. 3 is a flow chart explaining details of a method of assistinganalysis of a production process according to an embodiment of thepresent invention.

FIG. 4 is a flow chart explaining a specific example of a method ofassisting analysis of a production process according to an embodiment ofthe present invention.

FIG. 5 is a view of an example of comments shown in FIG. 4.

FIG. 6 is a view of an example of a screen including images displayed ina virtual space in the case where a user selects any three factors inthe specific example of FIG. 4.

FIG. 7 is a view of an example of a screen displaying images in athree-dimensional virtual space using a self-organizing map from four ormore dimensions of factors in the specific example of FIG. 4.

FIG. 8 is a view of a screen listing up and displaying images similar toa single image selected by a user in the specific example of FIG. 4.

FIG. 9 is a view of a screen displaying images having common factorsselected by a user in the specific example of FIG. 4.

FIG. 10 is a view of a screen where common image features are extractedby a user in the specific example of FIG. 4.

FIG. 11 is a view of a screen where common words are extracted by a userin the specific example of FIG. 4.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 is a flow chart explaining in brief the method of assistinganalysis of a production process according to an embodiment of thepresent invention. In the figure, the process data comprises the factors1 to L which may influence the quality of products in a productionprocess of products, product images 1 to M, and comments (text data) 1to Z relating to the process prepared by a production worker in theproduction process of products assigned to Product Nos. P₁ to P_(N).

In the illustrated example, the numeral/character data of the factors 1to L of Product No. P₁ are X11 to X1L, the images are I₁₁ to I_(1M), andthe comments are T₁₁ to T_(1z). The numeral/character data of thefactors 1 to L of Product No. P₂ are X₂₁ to X_(2L), the images are I₂₁to I_(2M), and the comments are T₂₁ to T_(2z). The numeral/characterdata of the factors 1 to L of Product No. P_(N) are X_(N1) to X_(NL),the images are I_(N1) to I_(NM), and the comments are T_(N1) to T_(NZ).

The images of the products are stored as image data in a storage deviceof a computer, while the comments are stored as text data in the storagedevice of the computer.

<Image Feature Extraction Routine S11>

At step S11, features are extracted from the image data of the imagesand converted to numerals/characters according to the present invention.The “numerals/characters” in the following description means a set ofsymbols comprised of at least one of the numerals/characters. As imagefeatures extracted, color features, texture features, frequency features(fourier features or DCT features), and shape features (seeabove-mentioned Nonpatent Documents 4 and 5). Further, it is alsopossible to automatically divide the images into pluralities of regions(segmentation) and extract the image features corresponding to theindividual regions or have a specific region designated by the user andextract the image features corresponding to that region.

<Text Feature Extraction Routine S12>

Similar, at step S12, features are extracted from the text data of thecomments and converted to numerals/characters according to the presentinvention. The text features are extracted as follows. First, a set ofwords believed to be effective for characterizing the text is selectedin advance. The tf-idf (term frequency-inverse document frequency)method is used to measure the relative importance of each word. Theseare listed to extract a vector having the t-idf values of the words aselements from the text data (see Nonpatent Document 6).

The image data converted to numerals/characters and the text dataconverted to numerals/characters are added to the process data asfactors other than factors 1 to L.

The process data is displayed on a display device connected to acomputer. A user designates at least one factor in the displayed processdata by clicking on it by the mouse. That is, the user designates atleast one factor among the image features, text features, and thefactors 1 to L of the product for each product. The computer receivesthe designation of the factors.

<Image Arrangement Routine S13>

Next, at step S13, the computer arranges the image data of the productsin a virtual space of up to three dimensions in accordance with thedesignations. The numeral/character data expressing the factorsincluding image features and text features are generally higherdimension vectors. With this, arrangement in a virtual space of threedimensions or less would be impossible. Therefore, to determine thearrangement of images in for example a three-dimensional virtual space,there is the method of selecting three numerals/characters from thenumerical/word data of the products and assigning them to threeorthogonal coordinate axes set in the space. Further, there is also themethod of compressing image features, text features, and numerical/worddata to three dimensions by principal component analysis,multi-dimensional scaling analysis, or another statistical technique(see Nonpatent Document 7). Further, it is also possible to use a selforganizing map (SOM)—a type of neural network (see Nonpatent Document8).

<Virtual Space Display Routine S13>

Next, at step S14, the virtual space in which the images are arranged isactually displayed on the display device. The images can be displayed byutilizing computer graphic technology so that the user can change hisviewing point in the virtual space or walk freely through the virtualspace (walk through) or fly around it (fly through). This assists thework of viewing a large number of images or narrowing down to someimages or zooming into a specific image and examining it in detail.

Further, it is also possible to display the virtual space in anotherdevice through a network. In this case, the images are transferredthrough a client-server network.

Further, a user can utilize an image arrangement routine while movingthrough the virtual space. Specifically, he can extract new types offeatures or change the method of assignment of numerals/characters tothe coordinate axes or change the types of features for arrangement ofthe images. Due to this, it becomes possible to assist the work of theuser comparing image data from various viewing points and finding therelationships between the various factors and patterns of the imagedata.

The user repeats the designation reception routine, the arrangementroutine, and the display routine until the user judges that there issimilarity between adjoining images in the displayed images while movingthrough the virtual space.

<Similar Image Search Routine S15>

Next, at step S15, the user uses a mouse or keyboard to select similarimages or designate the range of adjoining similar images from theimages displayed in the virtual space. Further, the user may also selecta specific image by a mouse or keyboard or designate a feature ornumeral/character data to be noted in the images so as to list up anddisplay similar images. Due to this, it becomes possible to assist thework of a user comparing image data from various viewing points andfinding the relationship between the various factors and patterns of theimage data.

<Common Factor Extraction Routine S16>

Further, at step S16, the user can select a plurality of images by themouse or keyboard in the virtual space and further designate the type offeature to be noted and the numeral/character data to extract imagefeatures, words, and factors common to the products corresponding to theimages. Due to this, it becomes possible to assist the work of findingthe relationship between the factors and features used for thearrangement and the extracted image features, words, and factors.

<Relation Verification Routine S17>

Finally, at step S17, the method verifies the “relationship betweenfactors and quality/characteristics” found by the user (person analyzingprocess) by the above routine. For example, when finding the rule that“when a certain factor is in a specific range, a quality/characteristictends to fall in a certain range” as this relationship, the methodchecks whether this rule stands or not for a large amount of data toquantitatively verify the appropriateness of the rule. Further, whenfinding the rule that “a certain factor and quality/characteristic havecorrelation”, it checks to what extent this relationship stands for alarge amount of data to quantitatively verify the appropriateness of therelationship found.

According to the present invention, there are also provided a programfor making a computer execute this method, a program product, and astorage medium storing this program.

Note that the program and program product in the present specificationinclude a program and program product distributed through a network anda program and program product stored in a storage medium.

FIG. 2 is a flow chart explaining a routine adding image features andtext features as factors in the process data shown in FIG. 1. In thefigure, at step S21, it is judged if the processing for extraction ofimage features and text features has been completed for all product datain the process data. If not completed, at step S22, image features areextracted from the image data of one product and added to the processdata as factors.

Next, at step S23, text features are extracted from the text data of theproduct and added to the process data as factors.

Next, at step S24, numeral/character data of the product in the processdata are extracted and added to the process data as factors.

Next, the routine returns to step S21, where it is judged if theprocessing for extraction of image features and text features for allproduct data in the process data has been completed. If not completed,steps S22 to S24 are repeated. If completed, the routine for addingfactors to the process data is ended.

Note that the processing shown in FIG. 2 is not processing essential inthe present invention.

FIG. 3 is a flow chart explaining details of a method of assistinganalysis of a production process according to an embodiment of thepresent invention.

In the figure, at step S301, the computer receives designation offactors in the process data from the user by a mouse or keyboard.

Next, at step S302, it is judged if the designation of factors hasended. If there is no further designation of a factor from the user, theprocessing is ended.

If a factor is designated, the routine proceeds to step S303 where dataof arrangement in a virtual space in the computer is prepared based onthe designated factors.

Next, at step S304, the images in the product data are displayed in thevirtual space based on the arrangement data prepared at step S303. Atstep S305, designation of factors from a user is received. The routinethen returns to step S302, whereupon step S303 to step S305 are repeateduntil the user judges that there is similarity between the adjoiningimages. The above operation is the detailed operation from step S11 toS14 in FIG. 1.

Next, at step S306, designation of a plurality of images is received bymouse operation or keyboard operation of the user.

Next, at step S307, it is judged if images have been designated from theuser. If not, the processing is ended.

If images have been designated, at step S308, the computer automaticallytakes out and displays only the plurality of images similar to thedesignated images.

Next, at step S309, the computer automatically extracts subsets offactors common to the images taken out and displayed from the factorsdesignated by the user.

Next, at step S310, the computer receives a hypothesis prepared by theuser relating to the relationship between a factor designated by theuser and the quality of the products.

Next, at step S311, the computer displays and compares the relatedimages so that the user can verify the relation hypothesis received.

Next, at step S312, the computer again receives designation of at leastone image by the user, judges if an image has been designated from theuser at step S307, and, if there is designation, repeats steps S308 toS311.

FIG. 4 is a flow chart explaining a specific example of a method ofassisting analysis of a production process according to an embodiment ofthe present invention.

In the figure, routines the same as in FIG. 1 are assigned the samereference numerals.

In this specific example, the method assisting analysis of a productionprocess covering analysis of the process of “pouring concrete”comprising creating a box-shaped structure at the base part or wall partof a building, pouring concrete into it, and utilizing the properties ofconcrete for cure it. Here, the objective is to beautify the appearanceafter pouring and reduce variations due to error factors.

The flow of processing of this specific example is as follows:

(1) Acquisition of Data

As the control factors, the following five factors are used:

Factor 1: Slump (indicator of extent of softness of still not yetsolidified concrete), unit: cm

Factor 2: Pouring speed, unit: m³/h (time)

Factor 3: Pouring stopping pressure, unit: kg/cm²

Factor 4: Maximum aggregate size, unit: mm

Factor 5: Fine aggregate ratio, unit: %

Further, it is learned that the ambient conditions of the pouring siteinfluence the finished product, so these are designated as errorfactors.

Factor 6: Air temperature, unit: ° C.

Factor 7: Humidity, unit: %

The image data was as follows:

Image 1: Image of appearance after pouring (for simplification of theillustration, the greater the number of hatching lines in the image, thedarker the color of the image. In actuality, the influence of muddywater during the pouring, the influence of the speed of pouring theconcrete, the influence of rain, etc. are reflected in the image).

The text data was as follows:

Comment 1: Sentence Describing Results of Observation During Pouring

FIG. 5 is a view showing part of an example of Comments T₁, T₂, . . .T_(N) corresponding to the product nos. In the figure, the Comment T₁ ofProduct No. P₁ is “I had the mud washed off, but some muddy waterremained. I instructed that it be wiped away by a rag”. Further, theComment T₂ of Product No. P₂ is “We ended up pouring the concrete all atonce into the frame, whereupon the side pressure became too large andthe frame could not withstand it”. Further, the Comment T_(N) of ProductNo. P_(N) is “30 minutes after starting the pouring, it began to rain abit. We covered the part finished being poured with a plastic sheet . .. ”

The image feature extraction routine for extracting feature quantitiesconverted to numerals/characters from image data of step S11 in FIG. 4is as follows:

(A) Extraction of Color Distribution Feature (see Nonpatent Document 5)

1) The pixel values of the image data are converted from expressions inan RGB color space (r, g, b) to expressions in an HSI color space (h, s,i).

2) The HSI space is divided into N_h (number of divisions in H-axis)×N_s(number of divisions in S-axis)×N_i (number of divisions in I-axis)=Nnumber of blocks and the numbers of pixels included in the individualblocks are counted. The results are listed and converted to anN-dimension vector. This is made the feature quantity.

(B) Extraction of Wavelet Feature (see Nonpatent Document 2)

1) The image data is converted to wavelets.

2) The wavelet conversion coefficients obtained as a result areseparated into large shape parts of the image and fine pattern parts,contour parts, etc.

3) These are combined to obtain a vector. This is used as the featurequantity.

The wavelet conversion is conversion by spatial frequency conversionwhile storing the positional information of the images. The shapes ofthe objects on images can be described compactly. Images of differentsizes can be compared with the same scale.

(C) Extraction of Edge Direction Histogram (see Nonpatent Document 4)

1) The edges are extracted from the image data by a Sobel filter etc.

2) The directions of the extracted edges are made discrete and thefrequency counted for each direction.

3) The values obtaining by division by the number of edges of the imageas a whole are arranged to make a vector. This is used as the featurequantity.

In addition, it is possible to extract fourier features, DCT features,texture features, and various other image features from image data andutilize them in the form of vectors (see Nonpatent Document 9).

The text feature extraction routine for converting the featurequantities into numerals/characters from the text data of step S12 ofFIG. 4 is performed by a method such as the one explained below (seeNonpatent Document 6).

1) First, words are extracted from the text data.

2) The tf-idf (term frequency-inverse document frequency) method is usedto measure the relative importance of each word and list up the same. Avector having the tf-idf values of the words as elements is obtainedfrom the text. This is used as the feature quantity.

The image arrangement and three-dimensional virtual space displayroutines of steps S13 and S14 of FIG. 4 are performed by the followingmethod.

FIG. 6 is a view showing a virtual space displayed in a display devicewhen selecting three numerals/characters from the numeral/character dataof the factors 1 to 7, images 1 to M, and comments 1 to Z and assigningthem to the three orthogonal coordinate axes set in the virtual space.The user selects three numerals/characters in the numeral/character dataof the factors including the images and comments and assigns them to thethree orthogonal coordinate axes set in the virtual space. For example,the user selects the factor 1 (slump), factor 5 (fine aggregate ratio),and factor 6 (temperature) and assigns these to the axes. By using thevirtual space display means, the image data of the products are arrangedin the virtual space and displayed on a display device connected to acomputer.

FIG. 7 is a view of an example of a screen displaying images in athree-dimensional virtual space using a self-organizing map from four ormore dimensions of factors in the specific example of FIG. 4.

In the case of text feature or image feature, a self organizing map(SOM) (see Nonpatent Document 8) is used to enable images to be arrangedon a three-dimensional virtual space or two-dimensional virtual space(plane) and enable images with similar features of these images to bearranged gathered close together. The self organizing map is a type ofneural network based on a competitive learning model and transfers thedata in a higher dimensional space to a lower dimensional space. At thistime, it is possible to arrange data close in distance in the higherdimensional space as close as possible in the lower dimensional space aswell. The SOM processing is divided into two phases: a learning phaseand an arrangement phase. In the learning phase, cells are arrangedregularly on a virtual space, then vector values assigned to the cellsare updated based on input. As a result of the learning, the cells atclose positions have similar vector values. The arrangement phase isperformed by arrangement at positions of cells having the closest vectorvalues to the vector value covered by the arrangement based on thelearning results. With this invented method, the SOM processing isperformed for different types of features using sets of information forarrangement and the results of arrangement are held.

The virtual space display routine is used to arrange the image data ofthe products in the virtual space and display it on a display deviceconnected to a computer.

From the image data, the color distribution feature (ex. HSV: HueSaturation Value, vector obtained by dividing color space divided intobins, dividing the numbers of pixels contained in the individual bins bythe number of pixels of the image as a whole), the texture featureshowing the degree of fineness (vector obtained by weighting waveletconversion coefficient), frequency feature showing regularity (fourierfeature or DCT feature), and shape factor showing cracks etc. (vectorobtained by extracting edges from image, making directions of the samediscrete, counting the frequency, dividing the result by the number ofedges of the image as a whole) are extracted (see Nonpatent Documents 4and 5).

In the virtual space shown in FIG. 6 or FIG. 7, as shown in step S41 inFIG. 4, it is possible to walk through the three-dimensional virtualspace. That is, the user can change the viewing point by operation ofthe mouse in the three-dimensional virtual space displayed to search fordesired images and thereby search for information. By clicking on animage, it is also possible to access detailed information of the productcorresponding to that image.

Further, in the virtual space shown in FIG. 6 or FIG. 7, as shown instep S42 in FIG. 4, it is possible to change the method of arrangementof the images. That is, by switching the factors assigned to the axesfor the display, it is possible to classify and arrange the images fromvarious viewpoints. For example, it is possible to select a text featureas the feature for arrangement and arrange the images for display usingthe same. Further, it is possible to input some sort of keywords andchange the arrangement of the corresponding images for display so thatthe images are closer to the viewing point of the user the better theextent by which text attributes included in comments on the productsmatch with the keywords. This operation is repeated until the userjudges that there is similarity between adjoining images.

FIG. 8 is a view of a screen listing up and displaying images similar toa single image selected by a user in the specific example of FIG. 4. Inthis figure by selecting a specific image 81 in the virtual space andfurther designating feature quantity or numeral/character data to benoted, it is possible to list up and display the similar images 82 to89. Due to this, it becomes possible to assist the work of the usercomparing the image data from various viewing points and discover arelationship between various factors and patterns of the image data.

FIG. 9 is a view of a screen displaying images having common factorsselected by a user in the specific example shown in FIG. 4. In thefigure, by selecting a plurality of images in a virtual space whereadjoining images appear to have similarity and further designatingfeature quantities or numeral/character data to be noted, it is possibleto extract factors, words, or image features common to the productscorresponding to these images. Due to this, it becomes possible toassist the work of finding the relationship between factors or featurequantities used for the arrangement and the factors, words, and imagefeatures extracted. For example, as shown in FIG. 9, if using thefactors 1 to 7, image 1, and comment 1 for arrangement, it is possibleto extract common factors by the following routine:

-   -   The factors 1 to 7, image 1, and comment 1 are arranged to a        nine-dimensional vector to form a two-dimensional self        organizing map (plane in three-dimensional space).    -   The learned self organizing map is used to arrange the images in        a virtual three-dimensional space.    -   The user views the virtual three-dimensional space and finds        locations where images having a certain common property are        gathered and locations where they are arranged with regularity.        For example, in FIG. 9, images 91 to 99 with a certain color        unevenness are gathered together. These images are selected by        surrounding them by a desired color frame.    -   An apparatus compares the distribution of the factors in the        images as a whole and calculates if the distribution of the        factors of the images selected are biased. The biased ones are        extracted as common factors. In FIG. 9, the factor 1 and factor        3 are extracted as candidates for common factors.    -   In this way, the rule for example that “if the slump of FIG. 1        is 16±1 cm, color unevenness easily occurs” can be discovered.

FIG. 10 is a view of a screen where common text is used for extractionby a user in the specific example shown in FIG. 4.

If using a text feature for arrangement, for example, it is possible toextract common factors as follows:

-   -   A text feature is used to form a self-organizing map.    -   The learned self organizing map is used to arrange images in a        virtual three-dimensional space.    -   The user views the virtual three-dimensional space and finds        locations where images having a certain common property are        gathered and locations where they are arranged with regularity.        For example, in FIG. 1, images of blue colors are gathered        together. The user selects these images surrounded by a yellow        frame. In FIG. 10, “rain”, “sheet”, “mud”, and other words in        comments are extracted as candidates for common words.    -   As a rule, “if covering by a sheet due to rain or if the ground        becomes muddy, a blue color easily appears” can be found.

FIG. 11 is a view of a screen where common image features are extractedby a user in the specific example shown in FIG. 4.

If using the image features for arrangement, it is possible to extractcommon factors for example as follows:

-   -   The texture feature is used as an image feature to form a        self-organizing map.    -   The learned self organizing map is used to arrange images in a        virtual three-dimensional space.

The user views the virtual three-dimensional space and finds locationswhere images having a certain common property are gathered and locationswhere they are arranged with regularity. For example, in FIG. 11, imagesof orange colors are gathered together. The user selects these imagessurrounded by a yellow frame. As a common feature, componentscorresponding to the “direction” in the texture feature in a certainrange are extracted as candidates of the common feature.

-   -   As a rule, “if the direction component of the texture is in a        certain range, an orange color is easily carried” can be found.

Next, the relationship verification routine of step S17 of FIG. 4 isexecuted as follows.

The “relationship between a factor and quality/characteristic”discovered by the user in the above routine is verified. For example,when finding the rule that “when a certain factor is in a specificrange, a quality/characteristic tends to fall in a certain range” asthis relationship, the method checks whether this rule stands or not fora large amount of data to quantitatively verify the appropriateness ofthe rule by comparison against the factors, images, orquality/characteristics appearing in the comment. Further, when findingthe rule that “a certain factor and quality/characteristic havecorrelation”, it checks to what extent this relationship stands for alarge amount of data to quantitatively verify the appropriateness of therelationship found by comparison against the factors, images orquality/characteristic appearing in the comments.

Note that the effects of the present invention can be obtained even ifthe factors do not include image data converted to numerals/charactersor text data converted to numerals/characters.

INDUSTRIAL APPLICABILITY

As clear from the above explanation, if using the method of assistingprocess analysis of the present invention, the program for making acomputer execute that method, or a storage medium storing that program,even when there are few records of data obtained in the productionprocess (corresponding to the number of products or lots), it ispossible to repeat the proposal and verification of a hypothesis whileincorporating the knowhow and experience of the person performing theanalysis and thereby possible to analyze the “relationship betweenfactors and characteristics”. Further, in process analysis, it ispossible to directly utilize image data expressing the characteristicsof the finished product or characteristics of the intermediate productor text data relating to the process such as comments of productionworkers obtained in the production process.

1. A method of assisting analysis of a production process assisting workfor analyzing the relationship between factors which may influence thequality of products in a production process and which is comprised ofnumeral/text data and quality/characteristic of these products, themethod of assisting analysis of a production process causing executionof a designation reception routine receiving at least one designation ofa factor from a user at a computer, an arrangement routine arrangingimages corresponding to said image data related to said received factorin a virtual space in said computer, a display routine displaying thevirtual space in which said images are arranged in said display device,a routine repeating said designation reception routine, said arrangementroutine, and said display routine until the user judges that there issimilarity between adjoining images in the displayed images, an imagedesignation reception routine receiving designation of a plurality ofimages by the user from images having similarity between adjoiningimages, a common factor extraction routine for automatically extractinga common factor among factors of said designated images, a relationhypothesis reception routine receiving a hypothesis of a relationshipbetween said common factors designated by the user and quality ofproducts, and a routine verifying the relation hypothesis so as tothereby assist work for analyzing the relationship between factorsselected by the user and said product characteristic.
 2. A method ofassisting analysis of a production process as set forth in claim 1,further including a routine extracting from image data expressing aquality/characteristic of said products feature quantity converted to anumeral/character and adding it to one of the factors and a routineextracting from text data relating to the process prepared by aproduction worker feature quantity converted to a numeral/character andadding it to one of the factors and the common factor extraction routineincludes a routine extracting at least one of image features, words, andfactors common to a selected image.
 3. A method of assisting analysis ofa production process as set forth in claim 1, wherein the designatedimage reception routine includes a routine making a computer receive aspecific image designated by a user in said virtual space and saiddisplay routine includes a routine making the computer list up anddisplay on said display means images similar to a designated image whena user designates at least one of feature quantity or numeral and worddata of a designated image.
 4. A method of assisting analysis of aproduction process as set forth in claim 1, further wherein the routinefor verifying a hypothesized relationship includes a routine making thecomputer verify a relationship between factors andquality/characteristic which a user finds.
 5. A method of assistinganalysis of a production process as set forth in claim 1, wherein thearrangement routine includes a routine making the computer utilize aself-organizing map.
 6. A method of assisting analysis of a productionprocess as set forth in claim 1, wherein the display routine includes aroutine making the computer change a viewing point in a virtual space soas to assist the work of viewing images in a virtual space.
 7. A methodof assisting analysis of a production process as set forth in claim 1,wherein the display routine includes a routine making a computer displaya virtual space at a display device connected to another computerthrough a network.
 8. A program product for assisting analysis of aproduction process assisting work for analyzing the relationship betweenfactors which may influence the quality of products in a productionprocess and which is comprised of numeral/text data and aquality/characteristic of these products, the program product forassisting analysis of a production process causing a computer to executea designation reception routine receiving at least one designation of afactor from a user, an arrangement routine arranging imagescorresponding to said image data related to said received factor in avirtual space in a display device connected to said computer, a displayroutine displaying the virtual space in which said images are arrangedin said display device, a routine repeating said designation receptionroutine, said arrangement routine, and said display routine until theuser judges that there is similarity between adjoining images in thedisplayed images, an image designation reception routine receivingdesignation of a plurality of images by the user from images havingsimilarity between adjoining images, a common factor extraction routinefor automatically extracting a common factor among factors of saiddesignated images, a relation hypothesis reception routine receiving ahypothesis of a relationship between said common factors designated bythe user and a quality of a product, and a routine verifying therelation hypothesis so as to thereby assist work for analyzing therelationship between factors selected by the user and said productcharacteristic.
 9. A program product for assisting analysis of aproduction process as set forth in claim 8, further including a routinemaking a computer extract from image data expressingquality/characteristic of said products feature quantity converted to anumeral/character and adding it to one of the factors and a routineextracting from text data relating to the process prepared by aproduction worker feature quantity converted to numeral/character andadding it to one of the factors and the common factor extraction routineincludes a routine of extracting at least one of image features, words,and factors common to a selected image.
 10. A program product forassisting analysis of a production process as set forth in claim 8,wherein the designated image reception routine includes a routine makinga computer receive a specific image designated by a user in said virtualspace and said display routine includes a routine making the computerlist up and display on said display means images similar to a designatedimage when a user designates at least one of feature quantity or numeraland character data of a designated image.
 11. A program product forassisting analysis of a production process as set forth in claim 8,wherein the routine for verifying a hypothesized relationship includes aroutine making the computer verify a relationship between factors andquality/characteristic which a user finds.
 12. A program product forassisting analysis of a production process as set forth in claim 8,wherein the arrangement routine includes a routine making the computerutilize a self-organizing map.
 13. A program product for assistinganalysis of a production process as set forth in claim 8, wherein thedisplay routine includes a routine making a computer change a viewingpoint in a virtual space so as to assist the work of viewing images in avirtual space.
 14. A program product for assisting analysis of aproduction process as set forth in claim 8, wherein the display routineincludes a routine making a computer display a virtual space at adisplay device connected to another computer through a network.
 15. Arecording medium storing a program for assisting analysis of aproduction process assisting work for analyzing the relationship betweenfactors which may influence the quality of products in a productionprocess and which is comprised of numeral/text data and aquality/characteristic of these products, the recording medium storing aprogram for assisting analysis of a production process causing acomputer to execute a designation reception routine receiving at leastone designation of a factor from a user, an arrangement routinearranging images corresponding to said image data related to saidreceived factor in a virtual space in a display device connected to saidcomputer, a display routine displaying the virtual space in which saidimages are arranged in said display device, a routine repeating saiddesignation reception routine, said arrangement routine, and saiddisplay routine until the user judges that there is similarity betweenadjoining images in the displayed images, an image designation receptionroutine receiving designation of a plurality of images by the user fromimages having similarity between adjoining images, a common factorextraction routine for automatically extracting common factors amongfactors of said designated images, a relation hypothesis receptionroutine receiving a hypothesis of a relationship between said commonfactors designated by the user and quality of products, and a routineverifying the relation hypothesis so as to thereby assist work foranalyzing the relationship between factors selected by the user and saidproduct characteristic.
 16. A recording medium storing a program forassisting analysis of a production process as set forth in claim 15,further including a routine for making a computer extract from imagedata expressing a quality/characteristic of said products quantityconverted to a numeral/character and adding it to one of the factors anda routine for extracting from text data relating to the process preparedby a production worker feature quantity converted to a numeral/characterand adding it to one of the factors and the common factor extractionroutine includes a routine of extracting at least one of image features,words, and factors common to a selected image.
 17. A recording mediumstoring a program for assisting analysis of a production process as setforth in claim 16, wherein the designated image reception routineincludes a routine making a computer receive a specific image designatedby a user in said virtual space and said display routine includes aroutine making the computer list up and display on said display meansimages similar to a designated image when a user designates at least oneof feature quantity or numeral and word data of a designated image. 18.A recording medium storing a program for assisting analysis of aproduction process as set forth in claim 15, further wherein the routinefor verifying a hypothesized relationship includes a routine making thecomputer verify a relationship between a factor andquality/characteristic which a user finds.
 19. A recording mediumstoring a program for assisting analysis of a production process as setforth in claim 15, wherein the arrangement routine includes a routinemaking the computer utilize a self-organizing map.
 20. A recordingmedium storing a program for assisting analysis of a production processas set forth in claim 15, wherein the display routine includes a routinemaking a computer change a viewing point in a virtual space so as toassist the work of viewing images in a virtual space.
 21. A recordingmedium storing a program for assisting analysis of a production processas set forth in claim 15, wherein the display routine includes a routinemaking a computer display a virtual space at a display device connectedto another computer through a network.